ChIP AML PiPeline v2

In [2]:
import os
from IPython.display import IFrame
import pandas as pd
import sys
sys.path.insert(0, '../..')
from JKBio.epigenetics import ChIP_helper as chiphelper
from JKBio import Helper as helper
import igv
import numpy as np
import pyBigWig
import itertools
import seaborn as sns
from scipy import stats
from matplotlib import cm
from scipy.cluster.hierarchy import linkage, leaves_list
from matplotlib import pyplot as plt
from sklearn.cluster import AgglomerativeClustering
from sklearn.mixture import GaussianMixture
from bokeh.plotting import *
from sklearn.manifold import MDS, TSNE
from sklearn.decomposition import PCA
from sklearn.preprocessing import scale
from pybedtools import BedTool
output_notebook()
%load_ext autoreload
%autoreload 2
Loading BokehJS ...

adding the data bucket to path

In [ ]:
! gcsfuse --only-dir Chip_AML jkobject data/seqs

processing using Nextflow

In [ ]:
singleend, pairedend = chiphelper.extractPairedSingleEndFrom('data/seqs')

Pipeline

  • Raw read QC (FastQC)
  • Adapter trimming (Trim Galore!)
  • Alignment (BWA)
  • Mark duplicates (picard)
  • Merge alignments from multiple libraries of the same sample (picard)
  • Re-mark duplicates (picard)
  • Filtering to remove: blacklisted regions, duplicates, primary alignments,unmapped,multiple locations, containing > 4 mismatches, insert size > 2kb, map to different chromosomes
  • Alignment-level QC and estimation of library complexity (picard, Preseq)
  • Create normalised bigWig files scaled to 1 million mapped reads (BEDTools, bedGraphToBigWig)
  • Generate gene-body meta-profile from bigWig files (deepTools)
  • Calculate genome-wide IP enrichment relative to control (deepTools)
  • Calculate strand cross-correlation peak and ChIP-seq quality measures including NSC and RSC (phantompeakqualtools)
  • Call broad/narrow peaks (MACS2)
  • Annotate peaks relative to gene features (HOMER)
  • Create consensus peakset across all samples and create tabular file to aid in the filtering of the data (BEDTools)
  • Count reads in consensus peaks (featureCounts)

In [ ]:
! nextflow cloud create 'JKcluster' -c 4
In [ ]:
! nextflow cloud create jkcluster -c "nextflow/nextflow.config" 40 && \
nextflow nf-core/chipseq -c "nextflow/nextflow.config" \
--singleEnd \
--seq_center 'DFCI' \
--email 'jkobject@gmail.com' \
--bucket-dir 'gs://jkobject/Chip_AML/nextflow/CHIPprocess_2/' \
--keyfile '~/jkobject-b6f1adaffcb8.json' \
--projectname 'jkobject' \
--zone 'us-east1-b' \
--skipDiffAnalysis \
--narrowPeak \
--design "nextflow/design.csv" \ 
--genome 'GRCh38' \
--profile gcp \
--resume \
--skipPreseq \
--max_cpus 8 && \
nextflow cloud shutdown jkclustert

Gathering data

In [ ]:
!gsutil -m cp -r gs://amlproject/Chip/results/bwa/mergedLibrary/macs/narrowPeak/ ../data
In [ ]:
!gsutil -m cp -r gs://amlproject/Chip/results/bwa/mergedLibrary/bigwig/ ../../data
In [ ]:
!cp ../../data/narrowPeak/*MV411*.narrowPeak ../../data/MV4narrow
In [110]:
! mkdir ../../data/BroadPeaks/MV411 && mv ../../data/BroadPeaks/MV411_* ../../data/BroadPeaks/MV411/
mv: cannot stat '../../data/BroadPeaks/MV411_*': No such file or directory
In [26]:
bindings = chiphelper.loadPeaks('../../data/MV4narrow/', isMacs=False,skiprows=0)
In [27]:
broadbindings = chiphelper.loadPeaks('../../data/BroadPeaks/MV411/', isMacs=False,skiprows=0)
In [28]:
SEgenes = pd.read_csv('../data/SEgenes.csv')
CTF = pd.read_csv('../data/CTF.csv', header=None)[0].tolist()
In [29]:
CTF.extend(['GATA2','IKZF1','LYL1' ,'PU1','SMC1'])
CTF
Out[29]:
['ARID2',
 'CEBPA',
 'CEBPE',
 'E2F3',
 'FLI1',
 'FOSL2',
 'GFI1',
 'GFI1B',
 'HHEX',
 'IRF8',
 'LYL1',
 'MEF2C',
 'MEF2D',
 'MEIS1',
 'MTF1',
 'MYB',
 'MYC',
 'PLAGL2',
 'RUNX1',
 'RUNX2',
 'RXRA',
 'SETDB1',
 'SNAPC5',
 'SP1',
 'SPI1',
 'SREBF1',
 'STAT5B',
 'TERF2',
 'TFAP4',
 'ZEB2',
 'ZFPM1',
 'ZMYND8',
 'LMO2',
 'MAX',
 'ELF2',
 'ETV6',
 'HOXA9',
 'GATA2',
 'GATA2',
 'IKZF1',
 'LYL1',
 'PU1',
 'SMC1']
In [31]:
CTF = list(set(CTF))
In [32]:
peaks = !ls ../../data/MV4narrow/*.narrowPeak
broadpeaks = ! ls ../../data/BroadPeaks/MV411/*.broadPeak
peaks = set([i.split('/')[-1].split('.')[0] for i in broadpeaks]) | set([i.split('/')[-1].split('.')[0] for i in peaks])

looking at the data and renaming

In [33]:
peaks
Out[33]:
{'mp106-MV411-MYB-r1',
 'mp112-MV411-CEBPA-r1',
 'mp115-MV411-PU1-r1',
 'mp116-MV411-MYB-r2',
 'mp117-MV411-POLII-r1',
 'mp118-MV411-SP1-r1',
 'mp129-MV411-RUNX1-r1',
 'mp136-MV411-H3K27ac-r1',
 'mp137-MV411-H3K4me3-r1',
 'mp138-MV411-H3K36me3-r1',
 'mp183-MV411_DMSO-H3K27ac-r1',
 'mp293-MV411-H3K79me2-r1',
 'mp299-MV411-CEBPA-r2',
 'mp300-MV411-RUNX1-r2',
 'mp301-MV411-PU1-r2',
 'mp319-MV411-FLI1-r1',
 'mp320-MV411-ZEB2-r1',
 'mp321-MV411-GFI1-r1',
 'mp324-MV411-MEF2D-r1',
 'mp325-MV411-SP1-r2',
 'mp357-MV411-ZMYND8-r1',
 'mp358-MV411-ZMYND8-r2',
 'mp359-MV411-LMO2-r1',
 'mp424-MV411-RUNX1-r3',
 'mp425-MV411-ETV6-r1',
 'mp426-MV411-SP1-r3',
 'mp427-MV411-LMO2-r2',
 'mp428-MV411-PU1-r3',
 'mp429-MV411-IRF8-r1',
 'mp430-MV411-IRF2BP2-r1',
 'mp431-MV411-IRF2BP2-r2',
 'mp433-MV411-GSE1-r1',
 'mp435-MV411-MED1-r1',
 'mp565-MV411-CDK13-r1',
 'mp568-MV411-ZEB2-r2',
 'mp569-MV411-ETV6-r2',
 'mp570-MV411-CTCF-r1',
 'mp575-MV411-IRF2BP2-r3',
 'mp577-MV411-SMC1-r1',
 'mp581-MV411-MEF2C-r1',
 'mp582-MV411-MEIS1-r1',
 'mp583-MV411-ELF2-r1',
 'mp584-MV411-MYC-r1',
 'mp585-MV411-MYC-r2',
 'mp587-MV411-IKZF1-r1',
 'mp589-MV411-ETV6-r3',
 'mp590-MV411-GATA2-r1',
 'mp601-MV411-RUNX2-r1',
 'mp604-MV411-IKZF1-r2',
 'mp607-MV411-ELF2-r2',
 'mp614-MV411-MEF2C-r2',
 'mp623-MV411-MEF2D-r2',
 'mp627-MV411-MEF2C-r3',
 'mp639-MV411-MAX-r1',
 'mp640-MV411-LYL1-r1',
 'mp650-MV411-LDB1-r1',
 'mp690-MV411_DMSO-MAX-r1',
 'mp70-MV411-H3K27ac-r2',
 'mp702-MV411_DMSO-H3K27ac-r1',
 'mp71-MV411-H3K27me3-r1',
 'mp714-MV411-ZEB2-r3',
 'mp715-MV411-ZEB2-r4',
 'mp716-MV411-GSE1-r2',
 'mp717-MV411-GATA2-r2',
 'mp718-MV411-IRF2BP2-r4',
 'mp72-MV411-CEBPA-r3',
 'mp724-MV411_DMSO-MYC-r1',
 'mp728-MV411_DMSO-CEBPA-r1',
 'mp734-MV411_DMSO-H3K27ac-r1',
 'mp760-MV411_MEF2D_NT_SC_63-MEF2D_FLAG-r1',
 'mp761-MV411_PU1_NT_SC_1-PU1_FLAG-r1',
 'mp762-MV411_IRF2BP2_CT_GFP_8-IRF2BP2_FLAG-r1',
 'mp763-MV411-RUNX2-r2',
 'mp764-MV411-RUNX2-r3',
 'mp765-MV411-RUNX2-r4',
 'mp767-MV411-FOXP1-r1',
 'mp768-MV411-FOXP1-r2',
 'mp770-MV411-MYBL2-r1',
 'mp771-MV411-JUND-r1',
 'mp772-MV411-JUND-r2',
 'mp773-MV411-HOXA9-r1',
 'mp774-MV411-HOXA9-r2',
 'mp88-MV411-H3K27ac-r3',
 'mp89-MV411-H3K36me3-r2'}
In [34]:
set(bindings['name'])
Out[34]:
{'mp106-MV411-MYB-r1',
 'mp112-MV411-CEBPA-r1',
 'mp115-MV411-PU1-r1',
 'mp116-MV411-MYB-r2',
 'mp117-MV411-POLII-r1',
 'mp118-MV411-SP1-r1',
 'mp129-MV411-RUNX1-r1',
 'mp136-MV411-H3K27ac-r1',
 'mp137-MV411-H3K4me3-r1',
 'mp138-MV411-H3K36me3-r1',
 'mp183-MV411_DMSO-H3K27ac-r1',
 'mp293-MV411-H3K79me2-r1',
 'mp299-MV411-CEBPA-r2',
 'mp300-MV411-RUNX1-r2',
 'mp301-MV411-PU1-r2',
 'mp319-MV411-FLI1-r1',
 'mp320-MV411-ZEB2-r1',
 'mp321-MV411-GFI1-r1',
 'mp324-MV411-MEF2D-r1',
 'mp325-MV411-SP1-r2',
 'mp357-MV411-ZMYND8-r1',
 'mp358-MV411-ZMYND8-r2',
 'mp359-MV411-LMO2-r1',
 'mp424-MV411-RUNX1-r3',
 'mp425-MV411-ETV6-r1',
 'mp426-MV411-SP1-r3',
 'mp427-MV411-LMO2-r2',
 'mp428-MV411-PU1-r3',
 'mp429-MV411-IRF8-r1',
 'mp430-MV411-IRF2BP2-r1',
 'mp431-MV411-IRF2BP2-r2',
 'mp433-MV411-GSE1-r1',
 'mp435-MV411-MED1-r1',
 'mp565-MV411-CDK13-r1',
 'mp568-MV411-ZEB2-r2',
 'mp569-MV411-ETV6-r2',
 'mp570-MV411-CTCF-r1',
 'mp575-MV411-IRF2BP2-r3',
 'mp577-MV411-SMC1-r1',
 'mp581-MV411-MEF2C-r1',
 'mp582-MV411-MEIS1-r1',
 'mp583-MV411-ELF2-r1',
 'mp584-MV411-MYC-r1',
 'mp585-MV411-MYC-r2',
 'mp587-MV411-IKZF1-r1',
 'mp589-MV411-ETV6-r3',
 'mp590-MV411-GATA2-r1',
 'mp601-MV411-RUNX2-r1',
 'mp604-MV411-IKZF1-r2',
 'mp607-MV411-ELF2-r2',
 'mp614-MV411-MEF2C-r2',
 'mp623-MV411-MEF2D-r2',
 'mp627-MV411-MEF2C-r3',
 'mp639-MV411-MAX-r1',
 'mp640-MV411-LYL1-r1',
 'mp650-MV411-LDB1-r1',
 'mp690-MV411_DMSO-MAX-r1',
 'mp70-MV411-H3K27ac-r2',
 'mp702-MV411_DMSO-H3K27ac-r1',
 'mp71-MV411-H3K27me3-r1',
 'mp714-MV411-ZEB2-r3',
 'mp715-MV411-ZEB2-r4',
 'mp716-MV411-GSE1-r2',
 'mp717-MV411-GATA2-r2',
 'mp718-MV411-IRF2BP2-r4',
 'mp72-MV411-CEBPA-r3',
 'mp724-MV411_DMSO-MYC-r1',
 'mp728-MV411_DMSO-CEBPA-r1',
 'mp734-MV411_DMSO-H3K27ac-r1',
 'mp760-MV411_MEF2D_NT_SC_63-MEF2D_FLAG-r1',
 'mp761-MV411_PU1_NT_SC_1-PU1_FLAG-r1',
 'mp762-MV411_IRF2BP2_CT_GFP_8-IRF2BP2_FLAG-r1',
 'mp763-MV411-RUNX2-r2',
 'mp764-MV411-RUNX2-r3',
 'mp765-MV411-RUNX2-r4',
 'mp767-MV411-FOXP1-r1',
 'mp768-MV411-FOXP1-r2',
 'mp770-MV411-MYBL2-r1',
 'mp771-MV411-JUND-r1',
 'mp772-MV411-JUND-r2',
 'mp773-MV411-HOXA9-r1',
 'mp774-MV411-HOXA9-r2',
 'mp88-MV411-H3K27ac-r3',
 'mp89-MV411-H3K36me3-r2'}
In [35]:
bindings
Out[35]:
chrom start end peak_number foldchange -log10pvalue -log10qvalue relative_summit_pos name
0 chr1 9871 10034 MV411_2_325_R1_peak_1 5.57109 8.70544 6.08151 156 mp325-MV411-SP1-r2
1 chr1 9936 10413 MV411_8_765_R1_peak_1 6.76346 16.39293 14.03933 235 mp765-MV411-RUNX2-r4
2 chr1 9949 10426 MV411_7_764_R1_peak_1 4.73735 10.22378 8.08562 236 mp764-MV411-RUNX2-r3
3 chr1 10058 10368 MV411_4_424_R1_peak_1 10.08329 22.67156 19.41437 144 mp424-MV411-RUNX1-r3
4 chr1 10087 10317 MV411_2_614_R1_peak_1 5.91491 10.68008 6.74843 102 mp614-MV411-MEF2C-r2
... ... ... ... ... ... ... ... ... ...
2766768 chrY 56858556 56858723 MV411_2_112_R1_peak_26816 4.44683 10.80341 7.72958 122 mp112-MV411-CEBPA-r1
2766769 chrY 56859773 56859965 MV411_3_299_R1_peak_193743 3.21570 8.50020 6.46514 96 mp299-MV411-CEBPA-r2
2766770 chrY 56876514 56876695 MV411_2_429_R1_peak_66896 4.82272 7.79657 5.37607 56 mp429-MV411-IRF8-r1
2766771 chrY 56876539 56876697 MV411_3_428_R1_peak_117301 5.07913 7.33086 5.12118 25 mp428-MV411-PU1-r3
2766772 chrY 56878587 56878844 MV411_1_767_R1_peak_46797 4.21133 9.22911 6.77967 81 mp767-MV411-FOXP1-r1

2766773 rows × 9 columns

In [36]:
broadbindings
Out[36]:
chrom start end peak_number foldchange -log10pvalue -log10qvalue name relative_summit_pos
0 chr1 10147 10335 MV411_2_88_R1_peak_1 4.14442 5.58136 3.31624 mp88-MV411-H3K27ac-r3 188
1 chr1 180754 181052 MV411_2_88_R1_peak_2 3.16232 4.18591 2.10093 mp88-MV411-H3K27ac-r3 298
2 chr1 181115 181547 MV411_1_639_R1_peak_1 3.85919 4.86421 2.87509 mp639-MV411-MAX-r1 432
3 chr1 181388 181563 MV411_9_718_R1_peak_1 5.08100 7.51516 3.99681 mp718-MV411-IRF2BP2-r4 175
4 chr1 181407 181602 MV411_2_584_R1_peak_1 4.56132 6.24043 3.85907 mp584-MV411-MYC-r1 195
... ... ... ... ... ... ... ... ... ...
800805 chrY 56763402 56763638 MV411_3_575_R1_peak_2363 2.94849 9.23765 4.85923 mp575-MV411-IRF2BP2-r3 236
800806 chrY 56763413 56763628 MV411_1_430_R1_peak_5248 2.41318 5.65123 2.06922 mp430-MV411-IRF2BP2-r1 215
800807 chrY 56822238 56822491 MV411_2_301_R1_peak_112556 3.47139 5.47547 3.32617 mp301-MV411-PU1-r2 253
800808 chrY 56865632 56865780 MV411_9_718_R1_peak_10307 4.01875 6.12669 2.77499 mp718-MV411-IRF2BP2-r4 148
800809 chrY 56876522 56876799 MV411_3_428_R1_peak_129305 3.52829 4.30314 2.30604 mp428-MV411-PU1-r3 277

800810 rows × 9 columns

In [38]:
bindings = bindings[~bindings.name.isin(set(broadbindings.name))]
In [40]:
bindings = bindings.append(broadbindings)
/home/jeremie/.local/lib/python3.7/site-packages/pandas/core/frame.py:7138: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version
of pandas will change to not sort by default.

To accept the future behavior, pass 'sort=False'.

To retain the current behavior and silence the warning, pass 'sort=True'.

  sort=sort,
In [41]:
len(bindings)
Out[41]:
2819569
In [42]:
bindings['replicate']= [i.split('-')[-1][-1] for i in bindings['name']]
bindings['tf'] = [i.split('-')[2] for i in bindings['name']]
In [43]:
bindings['peak_number'] = ['_'.join([i.split('_')[2],i.split('_')[5]]) for i in bindings['peak_number']]
In [44]:
bindings
Out[44]:
-log10pvalue -log10qvalue chrom end foldchange name peak_number relative_summit_pos start replicate tf
0 8.70544 6.08151 chr1 10034 5.57109 mp325-MV411-SP1-r2 325_1 156 9871 2 SP1
1 16.39293 14.03933 chr1 10413 6.76346 mp765-MV411-RUNX2-r4 765_1 235 9936 4 RUNX2
2 10.22378 8.08562 chr1 10426 4.73735 mp764-MV411-RUNX2-r3 764_1 236 9949 3 RUNX2
3 22.67156 19.41437 chr1 10368 10.08329 mp424-MV411-RUNX1-r3 424_1 144 10058 3 RUNX1
4 10.68008 6.74843 chr1 10317 5.91491 mp614-MV411-MEF2C-r2 614_1 102 10087 2 MEF2C
... ... ... ... ... ... ... ... ... ... ... ...
800805 9.23765 4.85923 chrY 56763638 2.94849 mp575-MV411-IRF2BP2-r3 575_2363 236 56763402 3 IRF2BP2
800806 5.65123 2.06922 chrY 56763628 2.41318 mp430-MV411-IRF2BP2-r1 430_5248 215 56763413 1 IRF2BP2
800807 5.47547 3.32617 chrY 56822491 3.47139 mp301-MV411-PU1-r2 301_112556 253 56822238 2 PU1
800808 6.12669 2.77499 chrY 56865780 4.01875 mp718-MV411-IRF2BP2-r4 718_10307 148 56865632 4 IRF2BP2
800809 4.30314 2.30604 chrY 56876799 3.52829 mp428-MV411-PU1-r3 428_129305 277 56876522 3 PU1

2819569 rows × 11 columns

In [45]:
bindings.to_csv('../temp/bindings.bed',sep='\t',index=False)
In [3]:
bindings= pd.read_csv('../temp/bindings.bed',sep='\t',header=None, index_col=None,
                     names=["-log10pvalue","-log10qvalue", "chrom", "end", "foldchange", "name", "peak_number", "relative_summit_pos", "start", "replicate","tf"])
In [276]:
bindings
Out[276]:
-log10pvalue -log10qvalue chrom end foldchange name peak_number relative_summit_pos start replicate tf
0 8.70544 6.08151 chr1 10034 5.57109 mp325-MV411-SP1-r2 325_1 156 9871 2 SP1
1 16.39293 14.03933 chr1 10413 6.76346 mp765-MV411-RUNX2-r4 765_1 235 9936 4 RUNX2
2 10.22378 8.08562 chr1 10426 4.73735 mp764-MV411-RUNX2-r3 764_1 236 9949 3 RUNX2
3 22.67156 19.41437 chr1 10368 10.08329 mp424-MV411-RUNX1-r3 424_1 144 10058 3 RUNX1
4 10.68008 6.74843 chr1 10317 5.91491 mp614-MV411-MEF2C-r2 614_1 102 10087 2 MEF2C
... ... ... ... ... ... ... ... ... ... ... ...
2819564 9.23765 4.85923 chrY 56763638 2.94849 mp575-MV411-IRF2BP2-r3 575_2363 236 56763402 3 IRF2BP2
2819565 5.65123 2.06922 chrY 56763628 2.41318 mp430-MV411-IRF2BP2-r1 430_5248 215 56763413 1 IRF2BP2
2819566 5.47547 3.32617 chrY 56822491 3.47139 mp301-MV411-PU1-r2 301_112556 253 56822238 2 PU1
2819567 6.12669 2.77499 chrY 56865780 4.01875 mp718-MV411-IRF2BP2-r4 718_10307 148 56865632 4 IRF2BP2
2819568 4.30314 2.30604 chrY 56876799 3.52829 mp428-MV411-PU1-r3 428_129305 277 56876522 3 PU1

2819569 rows × 11 columns

In [20]:
from gsheets import Sheets
sheets = Sheets.from_files('~/.client_secret.json', '~/.storage.json')
url="https://docs.google.com/spreadsheets/d/1yFLjYB1McU530JnLgL0QIMAKIkVl3kl0_LCHje2gk8U"
gsheet = sheets.get(url).sheets[2].to_frame()
gsheet
Out[20]:
id cell line replicate protein quality matching input name processed name previous name bigwig ... ratio to droso unique mapped reads(droso) scaling factor Total QC folderNarrow folderCompensated folderQC folderBroad folder Bigwig folder diffPeaks
0 mp100 U937 1 INPUT NaN NaN Y mp100-U937-INPUT-r1 NaN https://console.cloud.google.com/storage/brows... ... NaN NaN NaN https://storage.cloud.google.com/amlproject/Ch... https://console.cloud.google.com/storage/brows... NaN https://console.cloud.google.com/storage/brows... https://console.cloud.google.com/storage/brows... https://console.cloud.google.com/storage/brows... NaN
1 mp101 NOMO1 1 INPUT NaN NaN Y mp101-NOMO1-INPUT-r1 NaN https://console.cloud.google.com/storage/brows... ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
2 mp102 UT7 1 INPUT NaN NaN Y mp102-UT7-INPUT-r1 NaN https://console.cloud.google.com/storage/brows... ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
3 mp106 MV411 1 MYB x INPUT_MV411 Y mp106-MV411-MYB-r1 NaN https://console.cloud.google.com/storage/brows... ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
4 mp109 M6 1 CEBPA x INPUT_M6 Y mp109-M6-CEBPA-r1 NaN https://console.cloud.google.com/storage/brows... ... NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
281 mp853 MV411_IRF2BP_VHL_6h 5 MED1 NaN INPUT_MV411 NaN mp853-MV411_IRF2BP_VHL_6h-MED1-r5 20200302_8_MP7868_S58_R1_001 https://console.cloud.google.com/storage/brows... ... 0.033200 912539.0 0.902119 NaN NaN NaN NaN NaN NaN NaN
282 mp854 MV411_IRF2BP_VHL_6h 3 FLAG_IRF2BP2 NaN INPUT_MV411 NaN mp854-MV411_IRF2BP_VHL_6h-FLAG_IRF2BP2-r3 20200302_9_MP7868_S59_R1_001 https://console.cloud.google.com/storage/brows... ... 0.043080 920224.0 0.186965 NaN NaN NaN NaN NaN NaN NaN
283 mp855 MV411_IRF2BP_VHL_6h 3 POLII_total NaN INPUT_MV411 NaN mp855-MV411_IRF2BP_VHL_6h-POLII_total-r3 20200302_10_MP7868_S60_R1_001 https://console.cloud.google.com/storage/brows... ... 0.022331 601143.0 0.550032 NaN NaN NaN NaN NaN NaN NaN
284 mp856 MV411_IRF2BP_VHL_6h 3 POLII_S2 NaN INPUT_MV411 NaN mp856-MV411_IRF2BP_VHL_6h-POLII_S2-r3 20200302_11_MP7868_S61_R1_001 https://console.cloud.google.com/storage/brows... ... 0.021828 464172.0 0.604606 NaN NaN NaN NaN NaN NaN NaN
285 mp857 MV411_IRF2BP_VHL_6h 3 POLII_S5 NaN INPUT_MV411 NaN mp857-MV411_IRF2BP_VHL_6h-POLII_S5-r3 20200302_12_MP7868_S62_R1_001 https://console.cloud.google.com/storage/brows... ... 0.011859 251545.0 1.000000 NaN NaN NaN NaN NaN NaN NaN

286 rows × 29 columns

In [21]:
bw = ! ls ../../data/bigwig
bw
Out[21]:
['IGG_MV411_R1.mLb.clN.bigWig',
 'INPUT_MV411_R1.mLb.clN.bigWig',
 'mp106-MV411-MYB-r1.mLb.clN.bigWig',
 'mp112-MV411-CEBPA-r1.mLb.clN.bigWig',
 'mp115-MV411-PU1-r1.mLb.clN.bigWig',
 'mp116-MV411-MYB-r2.mLb.clN.bigWig',
 'mp117-MV411-POLII-r1.mLb.clN.bigWig',
 'mp118-MV411-SP1-r1.mLb.clN.bigWig',
 'mp129-MV411-RUNX1-r1.mLb.clN.bigWig',
 'mp136-MV411-H3K27ac-r1.mLb.clN.bigWig',
 'mp137-MV411-H3K4me3-r1.mLb.clN.bigWig',
 'mp138-MV411-H3K36me3-r1.mLb.clN.bigWig',
 'mp183-MV411_DMSO-H3K27ac-r1.mLb.clN.bigWig',
 'mp293-MV411-H3K79me2-r1.mLb.clN.bigWig',
 'mp299-MV411-CEBPA-r2.mLb.clN.bigWig',
 'mp300-MV411-RUNX1-r2.mLb.clN.bigWig',
 'mp301-MV411-PU1-r2.mLb.clN.bigWig',
 'mp319-MV411-FLI1-r1.mLb.clN.bigWig',
 'mp320-MV411-ZEB2-r1.mLb.clN.bigWig',
 'mp321-MV411-GFI1-r1.mLb.clN.bigWig',
 'mp324-MV411-MEF2D-r1.mLb.clN.bigWig',
 'mp325-MV411-SP1-r2.mLb.clN.bigWig',
 'mp357-MV411-ZMYND8-r1.mLb.clN.bigWig',
 'mp358-MV411-ZMYND8-r2.mLb.clN.bigWig',
 'mp359-MV411-LMO2-r1.mLb.clN.bigWig',
 'mp424-MV411-RUNX1-r3.mLb.clN.bigWig',
 'mp425-MV411-ETV6-r1.mLb.clN.bigWig',
 'mp426-MV411-SP1-r3.mLb.clN.bigWig',
 'mp427-MV411-LMO2-r2.mLb.clN.bigWig',
 'mp428-MV411-PU1-r3.mLb.clN.bigWig',
 'mp429-MV411-IRF8-r1.mLb.clN.bigWig',
 'mp430-MV411-IRF2BP2-r1.mLb.clN.bigWig',
 'mp431-MV411-IRF2BP2-r2.mLb.clN.bigWig',
 'mp433-MV411-GSE1-r1.mLb.clN.bigWig',
 'mp435-MV411-MED1-r1.mLb.clN.bigWig',
 'mp565-MV411-CDK13-r1.mLb.clN.bigWig',
 'mp568-MV411-ZEB2-r2.mLb.clN.bigWig',
 'mp569-MV411-ETV6-r2.mLb.clN.bigWig',
 'mp570-MV411-CTCF-r1.mLb.clN.bigWig',
 'mp575-MV411-IRF2BP2-r3.mLb.clN.bigWig',
 'mp577-MV411-SMC1-r1.mLb.clN.bigWig',
 'mp581-MV411-MEF2C-r1.mLb.clN.bigWig',
 'mp582-MV411-MEIS1-r1.mLb.clN.bigWig',
 'mp583-MV411-ELF2-r1.mLb.clN.bigWig',
 'mp584-MV411-MYC-r1.mLb.clN.bigWig',
 'mp585-MV411-MYC-r2.mLb.clN.bigWig',
 'mp587-MV411-IKZF1-r1.mLb.clN.bigWig',
 'mp589-MV411-ETV6-r3.mLb.clN.bigWig',
 'mp590-MV411-GATA2-r1.mLb.clN.bigWig',
 'mp601-MV411-RUNX2-r1.mLb.clN.bigWig',
 'mp604-MV411-IKZF1-r2.mLb.clN.bigWig',
 'mp607-MV411-ELF2-r2.mLb.clN.bigWig',
 'mp614-MV411-MEF2C-r2.mLb.clN.bigWig',
 'mp623-MV411-MEF2D-r2.mLb.clN.bigWig',
 'mp627-MV411-MEF2C-r3.mLb.clN.bigWig',
 'mp639-MV411-MAX-r1.mLb.clN.bigWig',
 'mp640-MV411-LYL1-r1.mLb.clN.bigWig',
 'mp650-MV411-LDB1-r1.mLb.clN.bigWig',
 'mp690-MV411_DMSO-MAX-r1.mLb.clN.bigWig',
 'mp702-MV411_DMSO-H3K27ac-r1.mLb.clN.bigWig',
 'mp70-MV411-H3K27ac-r2.mLb.clN.bigWig',
 'mp714-MV411-ZEB2-r3.mLb.clN.bigWig',
 'mp715-MV411-ZEB2-r4.mLb.clN.bigWig',
 'mp716-MV411-GSE1-r2.mLb.clN.bigWig',
 'mp717-MV411-GATA2-r2.mLb.clN.bigWig',
 'mp718-MV411-IRF2BP2-r4.mLb.clN.bigWig',
 'mp71-MV411-H3K27me3-r1.mLb.clN.bigWig',
 'mp724-MV411_DMSO-MYC-r1.mLb.clN.bigWig',
 'mp728-MV411_DMSO-CEBPA-r1.mLb.clN.bigWig',
 'mp72-MV411-CEBPA-r3.mLb.clN.bigWig',
 'mp734-MV411_DMSO-H3K27ac-r1.mLb.clN.bigWig',
 'mp760-MV411_MEF2D_NT_SC_63-FLAG_MEF2D-r1.mLb.clN.bigWig',
 'mp761-MV411_PU1_NT_SC_1-FLAG_PU1-r1.mLb.clN.bigWig',
 'mp762-MV411_IRF2BP2_CT_GFP_8-FLAG_IRF2BP2-r1.mLb.clN.bigWig',
 'mp763-MV411-RUNX2-r2.mLb.clN.bigWig',
 'mp764-MV411-RUNX2-r3.mLb.clN.bigWig',
 'mp765-MV411-RUNX2-r4.mLb.clN.bigWig',
 'mp767-MV411-FOXP1-r1.mLb.clN.bigWig',
 'mp768-MV411-FOXP1-r2.mLb.clN.bigWig',
 'mp770-MV411-MYBL2-r1.mLb.clN.bigWig',
 'mp771-MV411-JUND-r1.mLb.clN.bigWig',
 'mp772-MV411-JUND-r2.mLb.clN.bigWig',
 'mp773-MV411-HOXA9-r1.mLb.clN.bigWig',
 'mp774-MV411-HOXA9-r2.mLb.clN.bigWig',
 'mp88-MV411-H3K27ac-r3.mLb.clN.bigWig',
 'mp89-MV411-H3K36me3-r2.mLb.clN.bigWig']
In [46]:
len(set(bindings.name))
Out[46]:
84
In [25]:
len(bw)
Out[25]:
86
In [48]:
# ONE off
for i in bw[2:]:
    a = gsheet[gsheet.id=='mp'+i.split('_')[2]].name.values[0]
    i = '../../data/bigwig/'+i
    a = '../../data/bigwig/'+a+'.mLb.clN.bigWig'
    ! mv $i $a
    print(a)
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-48-5b9fcdae9fcb> in <module>
      1 # ONE off
      2 for i in bw[2:]:
----> 3     a = gsheet[gsheet.id=='mp'+i.split('_')[2]].name.values[0]
      4     i = '../../data/bigwig/'+i
      5     a = '../../data/bigwig/'+a+'.mLb.clN.bigWig'

IndexError: list index out of range
In [22]:
set(bindings.name)
Out[22]:
{'mp106-MV411-MYB-r1',
 'mp112-MV411-CEBPA-r1',
 'mp115-MV411-PU1-r1',
 'mp116-MV411-MYB-r2',
 'mp117-MV411-POLII-r1',
 'mp118-MV411-SP1-r1',
 'mp129-MV411-RUNX1-r1',
 'mp136-MV411-H3K27ac-r1',
 'mp137-MV411-H3K4me3-r1',
 'mp138-MV411-H3K36me3-r1',
 'mp183-MV411_DMSO-H3K27ac-r1',
 'mp293-MV411-H3K79me2-r1',
 'mp299-MV411-CEBPA-r2',
 'mp300-MV411-RUNX1-r2',
 'mp301-MV411-PU1-r2',
 'mp319-MV411-FLI1-r1',
 'mp320-MV411-ZEB2-r1',
 'mp321-MV411-GFI1-r1',
 'mp324-MV411-MEF2D-r1',
 'mp325-MV411-SP1-r2',
 'mp357-MV411-ZMYND8-r1',
 'mp358-MV411-ZMYND8-r2',
 'mp359-MV411-LMO2-r1',
 'mp424-MV411-RUNX1-r3',
 'mp425-MV411-ETV6-r1',
 'mp426-MV411-SP1-r3',
 'mp427-MV411-LMO2-r2',
 'mp428-MV411-PU1-r3',
 'mp429-MV411-IRF8-r1',
 'mp430-MV411-IRF2BP2-r1',
 'mp431-MV411-IRF2BP2-r2',
 'mp433-MV411-GSE1-r1',
 'mp435-MV411-MED1-r1',
 'mp565-MV411-CDK13-r1',
 'mp568-MV411-ZEB2-r2',
 'mp569-MV411-ETV6-r2',
 'mp570-MV411-CTCF-r1',
 'mp575-MV411-IRF2BP2-r3',
 'mp577-MV411-SMC1-r1',
 'mp581-MV411-MEF2C-r1',
 'mp582-MV411-MEIS1-r1',
 'mp583-MV411-ELF2-r1',
 'mp584-MV411-MYC-r1',
 'mp585-MV411-MYC-r2',
 'mp587-MV411-IKZF1-r1',
 'mp589-MV411-ETV6-r3',
 'mp590-MV411-GATA2-r1',
 'mp601-MV411-RUNX2-r1',
 'mp604-MV411-IKZF1-r2',
 'mp607-MV411-ELF2-r2',
 'mp614-MV411-MEF2C-r2',
 'mp623-MV411-MEF2D-r2',
 'mp627-MV411-MEF2C-r3',
 'mp639-MV411-MAX-r1',
 'mp640-MV411-LYL1-r1',
 'mp650-MV411-LDB1-r1',
 'mp690-MV411_DMSO-MAX-r1',
 'mp70-MV411-H3K27ac-r2',
 'mp702-MV411_DMSO-H3K27ac-r1',
 'mp71-MV411-H3K27me3-r1',
 'mp714-MV411-ZEB2-r3',
 'mp715-MV411-ZEB2-r4',
 'mp716-MV411-GSE1-r2',
 'mp717-MV411-GATA2-r2',
 'mp718-MV411-IRF2BP2-r4',
 'mp72-MV411-CEBPA-r3',
 'mp724-MV411_DMSO-MYC-r1',
 'mp728-MV411_DMSO-CEBPA-r1',
 'mp734-MV411_DMSO-H3K27ac-r1',
 'mp760-MV411_MEF2D_NT_SC_63-MEF2D_FLAG-r1',
 'mp761-MV411_PU1_NT_SC_1-PU1_FLAG-r1',
 'mp762-MV411_IRF2BP2_CT_GFP_8-IRF2BP2_FLAG-r1',
 'mp763-MV411-RUNX2-r2',
 'mp764-MV411-RUNX2-r3',
 'mp765-MV411-RUNX2-r4',
 'mp767-MV411-FOXP1-r1',
 'mp768-MV411-FOXP1-r2',
 'mp770-MV411-MYBL2-r1',
 'mp771-MV411-JUND-r1',
 'mp772-MV411-JUND-r2',
 'mp773-MV411-HOXA9-r1',
 'mp774-MV411-HOXA9-r2',
 'mp88-MV411-H3K27ac-r3',
 'mp89-MV411-H3K36me3-r2'}
In [ ]:
replicates = chiphelper.findReplicates(folder='data/seqs/results/bwa/', sep='_', namings='_R([0-9])',namepos=0)

Visual inspection of the features and and look at QCs

process:

look at all t with a very low frip score as noted by encode.

look at all peaks tracks together and see for location of intense co binding.

  • if we can discern peaks and if, for some reasons, some good peaks are not called by macs.
  • if looks good and we can see a lot of peaks.
  • if a lot of noise but seems consistent with replicates.
  • if just seems to have very few peaks.

Validate still but flag as potentially bad.

Else remove.

results:

In [4]:
bad=[
"mp168",
"mp129",
"mp128",
"mp773",
"mp774",
"mp575",
"mp614",
"mp714",
"mp433",
"mp156",
"mp650",
"mp604",
"mp27",
"mp627",
"mp117",
"mp771",
"mp118",
"mp431",
"mp430",
"mp324",
"mp565",
"mp569",
"mp125",
"mp627",
"mp568",
"mp427",
"mp124",
"mp716",
"mp581",
"mp589",
"mp321",
"mp601",
"mp745",
"mp772",
"mp770",
"mp590",
"mp623",
"mp718"]

merging duplicates

In [6]:
%matplotlib inline
mergedpeak, tomergebam, remove, ratiosofunique = chiphelper.mergeReplicatePeaks(bindings,'../../data/bigwig/',markedasbad=bad, window=150, mincov=4, doPlot=True, minKL=10, cov={}, use='poisson', MINOVERLAP=0.25,lookeverywhere=True, only='',saveloc='')
/!/ should only be passed peaks with at least one good replicate
merging H3K27ac peaks
finish first overlaps lookup
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn6()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [58264 17774 92422 55269 63133 49918]
main rep is: mp70-MV411-H3K27ac-r2
mp88-MV411-H3K27ac-r3
overlap: 46.18380904979334%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
4.566751159767021 59.99586699238654 []
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
3.9836066952883655 31.04046550469226
we have recovered 4454 peaks, equal to 4.728087215906076% of the peaks in main replicate
mp183-MV411_DMSO-H3K27ac-r1
overlap: 30.93311116400857%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
4.0431644289418145 71.2581424171932 []
enough overlap
we have recovered 11237 peaks, equal to 11.928494846236319% of the peaks in main replicate
not enough overlap to advice to merge the bams.
oldnew overlap: 37.062513932677305%
                            new overlap: 37.062513932677305%
mp734-MV411_DMSO-H3K27ac-r1
overlap: 28.418558351907553%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
0.15521763999621374 126.36395110755977 []
enough overlap
we have recovered 35561 peaks, equal to 37.749328577646146% of the peaks in main replicate
mp702-MV411_DMSO-H3K27ac-r1
overlap: 25.539373742182597%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
0.17103892143839006 101.35630092319843 []
enough overlap
we have recovered 52509 peaks, equal to 55.74026304894748% of the peaks in main replicate
mp136-MV411-H3K27ac-r1
overlap: 16.066520958213413%
not enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
5.893623734901306 66.31397446936982 []
we did not had enough initial overlap.
enough overlap
we have recovered 52810 peaks, equal to 56.05978578176916% of the peaks in main replicate
enough from small overlap to advice to merge the peaks
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn6()` is retained for backwards compatibility; use `venn()` instead
  warn((
H3K27ac 128335
merging JUND peaks
finish first overlaps lookup
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn2()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [ 396 3339]
no peaks were good enough quality
wrong TF: JUND
main rep is: mp772-MV411-JUND-r2
mp771-MV411-JUND-r1
overlap: 71.96969696969697%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
6.447604368774627 79.70097316269296 []
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
5.553420101735967 26.166475217490035
we have recovered 401 peaks, equal to 52.213541666666664% of the peaks in main replicate
merging MYB peaks
finish first overlaps lookup
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn2()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [26343 15340]
main rep is: mp106-MV411-MYB-r1
mp116-MV411-MYB-r2
overlap: 53.46391830846904%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
1.6217497332710058 52.193216210232755 []
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
1.8599135382961507 66.08142568799634
we have recovered 669 peaks, equal to 2.528822528822529% of the peaks in main replicate
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn2()` is retained for backwards compatibility; use `venn()` instead
  warn((
MYB 26455
merging GATA2 peaks
finish first overlaps lookup
found total peak for this replicate set: [ 2089 18108]
main rep is: mp717-MV411-GATA2-r2
mp590-MV411-GATA2-r1
overlap: 0.46940578749723877%
not enough overlap
not using this replicate from the peakmatrix
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn2()` is retained for backwards compatibility; use `venn()` instead
  warn((
GATA2 18108
merging HOXA9 peaks
finish first overlaps lookup
found total peak for this replicate set: [1253  351]
no peaks were good enough quality
wrong TF: HOXA9
main rep is: mp774-MV411-HOXA9-r2
mp773-MV411-HOXA9-r1
overlap: 55.55555555555556%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
5.616635313077553 69.07236248863505 []
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
7.4967794810971 38.29559308369546
we have recovered 108 peaks, equal to 27.1356783919598% of the peaks in main replicate
merging LMO2 peaks
finish first overlaps lookup
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn2()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [ 4276 18982]
main rep is: mp359-MV411-LMO2-r1
mp427-MV411-LMO2-r2
overlap: 16.33126119481614%
not enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
6.428105372122856 15.98414903939718 []
we did not had enough initial overlap.
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in multiply
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
9.917975744529173 59.029113963084804
we have recovered 74 peaks, equal to 0.38898233809924304% of the peaks in main replicate
not enough overlap to advice to merge the bams.
oldnew overlap: 16.499841955536823%
                            new overlap: 16.684188393608075%
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn2()` is retained for backwards compatibility; use `venn()` instead
  warn((
LMO2 19024
we only have one replicate for LYL1 .. pass
merging ETV6 peaks
finish first overlaps lookup
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn3()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [7182 3732 6750]
main rep is: mp425-MV411-ETV6-r1
mp589-MV411-ETV6-r3
overlap: 17.74814814814815%
not enough overlap
not using this replicate from the peakmatrix
mp569-MV411-ETV6-r2
overlap: 41.88148148148148%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
7.917139199532557 152.28127083248265 []
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
11.52933130168157 30.10431222953313
we have recovered 33 peaks, equal to 0.48859934853420195% of the peaks in main replicate
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn3()` is retained for backwards compatibility; use `venn()` instead
  warn((
ETV6 6754
merging MEF2C peaks
finish first overlaps lookup
found total peak for this replicate set: [3571 3314 1811]
no peaks were good enough quality
wrong TF: MEF2C
main rep is: mp627-MV411-MEF2C-r3
mp581-MV411-MEF2C-r1
overlap: 84.98067366096079%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
1.111457451589632 70.81722184291493 []
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
6.469333753967152 13.84877848055114
we have recovered 266 peaks, equal to 13.768115942028986% of the peaks in main replicate
mp614-MV411-MEF2C-r2
overlap: 77.91275538376587%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
6.134014378741286 73.15781831306433 []
enough overlap
we have recovered 390 peaks, equal to 20.18633540372671% of the peaks in main replicate
merging MEF2D peaks
finish first overlaps lookup
!TF has less than 1000 PEAKS!
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn2()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [ 99 242]
no peaks were good enough quality
wrong TF: MEF2D
main rep is: mp623-MV411-MEF2D-r2
mp324-MV411-MEF2D-r1
overlap: 26.262626262626267%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
8.208587603073378 13.853039087055434 []
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
7.9227475226570725 30.800091255147418
we have recovered 44 peaks, equal to 31.428571428571427% of the peaks in main replicate
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
we only have one replicate for IRF2BP2_FLAG .. pass
we only have one replicate for MED1 .. pass
merging RUNX2 peaks
finish first overlaps lookup
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn4()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [70753 69206   470 75773]
main rep is: mp764-MV411-RUNX2-r3
mp763-MV411-RUNX2-r2
overlap: 84.52219128185502%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
0.7196078886424635 82.36721645920606 []
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
0.6342788987161722 96.87099533805838
we have recovered 9996 peaks, equal to 12.43670295489891% of the peaks in main replicate
mp765-MV411-RUNX2-r4
overlap: 81.67025193670568%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
0.9561759992880259 82.77871117095 []
enough overlap
we have recovered 12852 peaks, equal to 15.9900466562986% of the peaks in main replicate
mp601-MV411-RUNX2-r1
overlap: 0.5978382801261662%
not enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
9.295258862699995 42.36290990891203 []
we did not had enough initial overlap.
enough overlap
we have recovered 12862 peaks, equal to 16.00248833592535% of the peaks in main replicate
enough from small overlap to advice to merge the peaks
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn4()` is retained for backwards compatibility; use `venn()` instead
  warn((
RUNX2 83278
we only have one replicate for POLII .. pass
we only have one replicate for GFI1 .. pass
merging MYC peaks
finish first overlaps lookup
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn3()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [46937 22051 32121]
main rep is: mp584-MV411-MYC-r1
mp724-MV411_DMSO-MYC-r1
overlap: 57.60700513454204%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
5.561909418907015 75.35384888520299 []
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
0.5366007178714792 116.62135367600375
we have recovered 1920 peaks, equal to 3.9498045669615305% of the peaks in main replicate
mp585-MV411-MYC-r2
overlap: 42.50165114941304%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
7.351515257846612 25.936917199782712 []
enough overlap
we have recovered 1960 peaks, equal to 4.032092162106562% of the peaks in main replicate
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn3()` is retained for backwards compatibility; use `venn()` instead
  warn((
MYC 48869
we only have one replicate for FLI1 .. pass
we only have one replicate for IRF8 .. pass
merging IKZF1 peaks
finish first overlaps lookup
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn2()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [ 2429 36145]
main rep is: mp587-MV411-IKZF1-r1
mp604-MV411-IKZF1-r2
overlap: 6.4988241803845614%
not enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
6.777888672671017 30.58047930489978 []
we did not had enough initial overlap.
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
13.007377917857431 113.94384245604697
we have recovered 221 peaks, equal to 0.6104297867638935% of the peaks in main replicate
enough from small overlap to advice to merge the peaks
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn2()` is retained for backwards compatibility; use `venn()` instead
  warn((
IKZF1 36204
merging PU1 peaks
finish first overlaps lookup
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn3()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [  9325 128812 111479]
main rep is: mp428-MV411-PU1-r3
mp301-MV411-PU1-r2
overlap: 69.31264167934664%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
0.4358339084557014 159.04109771999094 []
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
0.3210624373993328 159.20566713198377
we have recovered 8956 peaks, equal to 6.543005135922969% of the peaks in main replicate
mp115-MV411-PU1-r1
overlap: 7.04903269881688%
not enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
15.893177484194874 29.407618109102074 []
we did not had enough initial overlap.
enough overlap
we have recovered 8959 peaks, equal to 6.5451968526947155% of the peaks in main replicate
enough from small overlap to advice to merge the peaks
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn3()` is retained for backwards compatibility; use `venn()` instead
  warn((
PU1 136942
merging ZMYND8 peaks
finish first overlaps lookup
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn2()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [ 7804 22889]
main rep is: mp357-MV411-ZMYND8-r1
mp358-MV411-ZMYND8-r2
overlap: 29.05325702302416%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
6.141529978669944 23.58823795843151 []
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
4.59371136519243 56.3274184418172
we have recovered 313 peaks, equal to 1.3534549857303468% of the peaks in main replicate
enough from small overlap to advice to merge the peaks
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn2()` is retained for backwards compatibility; use `venn()` instead
  warn((
ZMYND8 23126
merging ELF2 peaks
finish first overlaps lookup
found total peak for this replicate set: [ 5220 43262]
main rep is: mp583-MV411-ELF2-r1
mp607-MV411-ELF2-r2
overlap: 11.744718228468402%
not enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
8.660037108107248 46.186857002214666 []
we did not had enough initial overlap.
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
0.9465838259926627 191.78091218569563
we have recovered 70 peaks, equal to 0.16160310277957338% of the peaks in main replicate
enough from small overlap to advice to merge the peaks
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn2()` is retained for backwards compatibility; use `venn()` instead
  warn((
ELF2 43316
we only have one replicate for SMC1 .. pass
we only have one replicate for MEIS1 .. pass
merging ZEB2 peaks
finish first overlaps lookup
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn4()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [14823   862  6268  2375]
main rep is: mp715-MV411-ZEB2-r4
mp320-MV411-ZEB2-r1
overlap: 30.47965998785671%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in multiply
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
0.5282218551997994 57.23591820760516 []
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
1.4891481661903925 123.58472899387124
we have recovered 1449 peaks, equal to 9.680005344378381% of the peaks in main replicate
mp714-MV411-ZEB2-r3
overlap: 14.362814544963905%
not enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
1.0501958629598303 60.518031100427045 []
enough overlap
we have recovered 4190 peaks, equal to 27.991181775669716% of the peaks in main replicate
enough from small overlap to advice to merge the peaks
mp568-MV411-ZEB2-r2
overlap: 1.841732442825339%
not enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
we did not had enough initial overlap.
enough overlap
we have recovered 4190 peaks, equal to 27.991181775669716% of the peaks in main replicate
not enough overlap to advice to merge the bams.
oldnew overlap: 2.0308637851559888%
                            new overlap: 2.0308637851559888%
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn4()` is retained for backwards compatibility; use `venn()` instead
  warn((
ZEB2 14969
we only have one replicate for CTCF .. pass
merging SP1 peaks
finish first overlaps lookup
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn3()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [11639 26220  2527]
main rep is: mp325-MV411-SP1-r2
mp426-MV411-SP1-r3
overlap: 42.353165522501904%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in multiply
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
5.881477669540173 144.22503590935503 []
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
0.7976536047545938 171.48645496721602
we have recovered 400 peaks, equal to 1.5080113100848256% of the peaks in main replicate
mp118-MV411-SP1-r1
overlap: 9.12280701754386%
not enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
10.426587146701502 14.660956341575323 []
we did not had enough initial overlap.
enough overlap
we have recovered 407 peaks, equal to 1.53440150801131% of the peaks in main replicate
enough from small overlap to advice to merge the peaks
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn3()` is retained for backwards compatibility; use `venn()` instead
  warn((
SP1 26525
we only have one replicate for MYBL2 .. pass
we only have one replicate for CDK13 .. pass
we only have one replicate for H3K27me3 .. pass
merging CEBPA peaks
finish first overlaps lookup
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn4()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [ 56671 191400  31280  26517]
main rep is: mp299-MV411-CEBPA-r2
mp72-MV411-CEBPA-r3
overlap: 28.529258098223615%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
6.152746782969362 53.410173862691714 []
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
0.31197820519098823 102.39042875267754
we have recovered 1196 peaks, equal to 0.6214601195115614% of the peaks in main replicate
enough from small overlap to advice to merge the peaks
mp728-MV411_DMSO-CEBPA-r1
overlap: 15.776384535005224%
not enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
8.064517310805096 8.833976613747126 []
we did not had enough initial overlap.
enough overlap
we have recovered 1198 peaks, equal to 0.6224993504806443% of the peaks in main replicate
enough from small overlap to advice to merge the peaks
mp112-MV411-CEBPA-r1
overlap: 13.635318704284222%
not enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
6.966794009968104 60.32500524641245 []
we did not had enough initial overlap.
enough overlap
we have recovered 1256 peaks, equal to 0.6526370485840478% of the peaks in main replicate
enough from small overlap to advice to merge the peaks
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn4()` is retained for backwards compatibility; use `venn()` instead
  warn((
CEBPA 192700
we only have one replicate for MAX .. pass
merging FOXP1 peaks
finish first overlaps lookup
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn2()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [32252 45923]
main rep is: mp767-MV411-FOXP1-r1
mp768-MV411-FOXP1-r2
overlap: 64.63863423556823%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
1.313552415345718 101.09725006794116 []
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
1.6468184971500788 82.57614260622178
we have recovered 2323 peaks, equal to 5.014354480108792% of the peaks in main replicate
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn2()` is retained for backwards compatibility; use `venn()` instead
  warn((
FOXP1 46327
merging GSE1 peaks
finish first overlaps lookup
!TF has less than 1000 PEAKS!
found total peak for this replicate set: [785 179]
no peaks were good enough quality
wrong TF: GSE1
main rep is: mp433-MV411-GSE1-r1
mp716-MV411-GSE1-r2
overlap: 73.18435754189943%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
6.8258870411537815 66.2275190278784
we have recovered 159 peaks, equal to 47.0414201183432% of the peaks in main replicate
merging IRF2BP2 peaks
finish first overlaps lookup
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn4()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [ 2324  4117  5211 10272]
no peaks were good enough quality
wrong TF: IRF2BP2
main rep is: mp575-MV411-IRF2BP2-r3
mp718-MV411-IRF2BP2-r4
overlap: 65.83476764199656%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
6.561096283218694 95.2656348730189 []
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
7.178695758636794 37.3130776115444
we have recovered 143 peaks, equal to 6.054191363251482% of the peaks in main replicate
mp430-MV411-IRF2BP2-r1
overlap: 57.31497418244407%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2644: RuntimeWarning: invalid value encountered in double_scalars
  if (fx2 - fval) > delta:
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2653: RuntimeWarning: invalid value encountered in double_scalars
  if 2.0 * (fx - fval) <= bnd:
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: invalid value encountered in true_divide
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
5.377302341753139 74.54312104621668 []
enough overlap
we have recovered 304 peaks, equal to 12.870448772226926% of the peaks in main replicate
mp431-MV411-IRF2BP2-r2
overlap: 52.49569707401033%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
5.577425461226521 39.09551385417364 []
enough overlap
we have recovered 447 peaks, equal to 18.92464013547841% of the peaks in main replicate
we only have one replicate for LDB1 .. pass
we only have one replicate for H3K79me2 .. pass
merging H3K36me3 peaks
finish first overlaps lookup
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn2()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [20895 94286]
main rep is: mp89-MV411-H3K36me3-r2
mp138-MV411-H3K36me3-r1
overlap: 1.8199944848651974%
not enough overlap
Wrong TF: H3K36me3
we only have one replicate for PU1_FLAG .. pass
we only have one replicate for MEF2D_FLAG .. pass
we only have one replicate for H3K4me3 .. pass
merging RUNX1 peaks
finish first overlaps lookup
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn3()` is retained for backwards compatibility; use `venn()` instead
  warn((
found total peak for this replicate set: [42806  2088 41413]
main rep is: mp300-MV411-RUNX1-r2
mp424-MV411-RUNX1-r3
overlap: 75.67397093865345%
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: divide by zero encountered in log
  bw = pyBigWig.open(filepath)
5.45480585254389 81.10407436970999 []
enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
5.1343263875435055 79.10158571081898
we have recovered 309 peaks, equal to 0.7187718073970691% of the peaks in main replicate
mp129-MV411-RUNX1-r1
overlap: 4.807737233098164%
not enough overlap
../../JKBio/epigenetics/ChIP_helper.py:780: RuntimeWarning: invalid value encountered in log
  bw = pyBigWig.open(filepath)
../../JKBio/epigenetics/ChIP_helper.py:778: RuntimeWarning: invalid value encountered in log
  def negLogLikelihood(params, data): return - np.sum(np.log(poisson(data, params[0])))
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in exp
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2371: RuntimeWarning: invalid value encountered in double_scalars
  w = xb - ((xb - xc) * tmp2 - (xb - xa) * tmp1) / denom
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1984: RuntimeWarning: invalid value encountered in double_scalars
  tmp1 = (x - w) * (fx - fv)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:1985: RuntimeWarning: invalid value encountered in double_scalars
  tmp2 = (x - v) * (fx - fw)
../../JKBio/epigenetics/ChIP_helper.py:779: RuntimeWarning: overflow encountered in power
  def poissonFit(data): return float(minimize(negLogLikelihood,x0=np.ones(1),args=(data,),method='Powell').x)
/home/jeremie/.local/lib/python3.8/site-packages/scipy/optimize/optimize.py:2672: RuntimeWarning: invalid value encountered in double_scalars
  t *= temp*temp
8.2009771553283 76.22653286177888 []
we did not had enough initial overlap.
enough overlap
we have recovered 328 peaks, equal to 0.7629681321237497% of the peaks in main replicate
enough from small overlap to advice to merge the peaks
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:15: UserWarning: `get_labels()` is retained for backwards compatibility; use `generate_petal_labels()` or the higher level `venn()` instead
  warn((
/home/jeremie/.local/lib/python3.8/site-packages/venn/_backwards_compatibility.py:30: UserWarning: `venn3()` is retained for backwards compatibility; use `venn()` instead
  warn((
RUNX1 42990
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-bb7ada4a99fe> in <module>
      1 get_ipython().run_line_magic('matplotlib', 'inline')
----> 2 mergedpeak, tomergebam, remove, ratiosofunique = chiphelper.mergeReplicatePeaks(bindings,'../../data/bigwig/',markedasbad=bad, window=150, mincov=4, doPlot=True, minKL=10, cov={}, use='poisson', MINOVERLAP=0.25,lookeverywhere=True, only='',saveloc='')

~/JKBio/epigenetics/ChIP_helper.py in mergeReplicatePeaks(peaks, bigwigfolder, markedasbad, window, sampling, mincov, doPlot, cov, minKL, use, MINOVERLAP, lookeverywhere, only, saveloc)
    752             fig = fig.map_upper(col_nan_scatter)
    753             fig = fig.map_upper(col_nan_kde_histo)
--> 754             plt.suptitle("correlation and distribution of peaks after recovery", y=1.08)
    755             if saveloc:
    756                 plt.savefig(saveloc+tf+"_after_pairplot.png")

~/.local/lib/python3.8/site-packages/pandas/core/frame.py in __init__(self, data, index, columns, dtype, copy)
    433             )
    434         elif isinstance(data, dict):
--> 435             mgr = init_dict(data, index, columns, dtype=dtype)
    436         elif isinstance(data, ma.MaskedArray):
    437             import numpy.ma.mrecords as mrecords

~/.local/lib/python3.8/site-packages/pandas/core/internals/construction.py in init_dict(data, index, columns, dtype)
    252             arr if not is_datetime64tz_dtype(arr) else arr.copy() for arr in arrays
    253         ]
--> 254     return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)
    255 
    256 

~/.local/lib/python3.8/site-packages/pandas/core/internals/construction.py in arrays_to_mgr(arrays, arr_names, index, columns, dtype)
     62     # figure out the index, if necessary
     63     if index is None:
---> 64         index = extract_index(arrays)
     65     else:
     66         index = ensure_index(index)

~/.local/lib/python3.8/site-packages/pandas/core/internals/construction.py in extract_index(data)
    353 
    354         if not indexes and not raw_lengths:
--> 355             raise ValueError("If using all scalar values, you must pass an index")
    356 
    357         if have_series:

ValueError: If using all scalar values, you must pass an index
In [ ]:
tomergebam
In [ ]:
mergedpeak = mergedpeak[mergedpeak.columns[[2,9,3,5,6,4,0,1,7,10]]]
In [ ]:
mergedpeak.to_csv('../temp/merged_replicates.csv')
In [7]:
mergedpeak = pd.read_csv('../temp/merged_replicates.csv', index_col=0)
/home/jeremie/.local/lib/python3.8/site-packages/IPython/core/interactiveshell.py:3062: DtypeWarning: Columns (5) have mixed types.Specify dtype option on import or set low_memory=False.
  has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
/home/jeremie/.local/lib/python3.8/site-packages/numpy/lib/arraysetops.py:569: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
  mask |= (ar1 == a)

show replicates overlap

sorting and removing samples

In [7]:
bigwigs=os.listdir('../../data/bigwig/')
for val in bigwigs:
    for v in remove + toremove + ['scale','POLII','IGG','CTCF','INPUT']:
        if v in val:
            bigwigs.remove(val)
            break
bigwigs = ['data/bigwig/'+ i for i in bigwigs]
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-7-4253376083d3> in <module>
      1 bigwigs=os.listdir('../../data/bigwig/')
      2 for val in bigwigs:
----> 3     for v in remove + toremove + ['scale','POLII','IGG','CTCF','INPUT']:
      4         if v in val:
      5             bigwigs.remove(val)

NameError: name 'remove' is not defined
In [62]:
set(mergedpeak.tf)
Out[62]:
{'CDK13',
 'CEBPA',
 'CTCF',
 'ELF2',
 'ETV6',
 'FLI1',
 'FOXP1',
 'GATA2',
 'GFI1',
 'H3K27ac',
 'H3K27me3',
 'H3K4me3',
 'H3K79me2',
 'IKZF1',
 'IRF2BP2_FLAG',
 'IRF8',
 'LDB1',
 'LMO2',
 'LYL1',
 'MAX',
 'MED1',
 'MEF2D_FLAG',
 'MEIS1',
 'MYB',
 'MYBL2',
 'MYC',
 'POLII',
 'PU1',
 'PU1_FLAG',
 'RUNX1',
 'RUNX2',
 'SMC1',
 'SP1',
 'ZEB2',
 'ZMYND8'}
In [63]:
mergedpeak.foldchange.min()
Out[63]:
1.1767
In [ ]:
mergedpeak['name']=mergedpeak.tf
In [ ]:
## Removing bad ChIP protein
mergedpeak = mergedpeak[~mergedpeak['name'].isin(['CDK13','GSE1'])]

Create a consensus set

In [ ]:
merged = chiphelper.simpleMergedPeaks(mergedpeak[~mergedpeak.tf.isin(['MED1','SMC1','CTCF','POLII','IRF2BP2_FLAG','IRF2BP2', 'H3K27ac', 'H3K27me3', 'H3K4me3', 'H3K79me2',])], window=150)
In [49]:
len(merged)
Out[49]:
401964
In [8]:
len(mergedpeak)
Out[8]:
1795655
In [53]:
merged
Out[53]:
chrom start end peak_number foldchange -log10pvalue -log10qvalue relative_summit_pos ZEB2 PU1 ... RUNX2 FOXP1 MYB ETV6 MAX MEIS1 FLI1 LYL1 MEF2D_FLAG ELF2
0 chr1 9871 10426 0 5.660748 167.597710 113.516687 236 0.0 0.00000 ... 5.750405 0.00000 0.0 0.0 0.00000 0.00000 0.0 0.0 0.0 0.0
1 chr1 181115 181583 1 5.582072 13.719270 10.515070 432 0.0 0.00000 ... 0.000000 0.00000 0.0 0.0 3.85919 6.42691 0.0 0.0 0.0 0.0
2 chr1 267795 268177 2 7.898975 448.796925 351.318955 200 0.0 0.00000 ... 7.898975 0.00000 0.0 0.0 0.00000 0.00000 0.0 0.0 0.0 0.0
3 chr1 271061 271390 3 5.825944 4270.465248 1288.072707 224 0.0 3.69205 ... 5.431833 0.00000 0.0 0.0 4.44630 0.00000 0.0 0.0 0.0 0.0
4 chr1 274361 274524 4 3.919940 4.925860 2.932390 163 0.0 0.00000 ... 0.000000 0.00000 0.0 0.0 3.91994 0.00000 0.0 0.0 0.0 0.0
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
401959 chrY 56853053 56853501 401959 8.004440 45.614230 43.286370 189 0.0 0.00000 ... 0.000000 0.00000 0.0 0.0 0.00000 0.00000 0.0 0.0 0.0 0.0
401960 chrY 56858512 56858723 401960 6.185595 519.925882 350.663218 154 0.0 0.00000 ... 0.000000 0.00000 0.0 0.0 0.00000 0.00000 0.0 0.0 0.0 0.0
401961 chrY 56859773 56859965 401961 3.215700 8.500200 6.465140 96 0.0 0.00000 ... 0.000000 0.00000 0.0 0.0 0.00000 0.00000 0.0 0.0 0.0 0.0
401962 chrY 56876514 56876799 401962 4.175505 7.796570 5.376070 56 0.0 3.52829 ... 0.000000 0.00000 0.0 0.0 0.00000 0.00000 0.0 0.0 0.0 0.0
401963 chrY 56878587 56878844 401963 4.211330 9.229110 6.779670 81 0.0 0.00000 ... 0.000000 4.21133 0.0 0.0 0.00000 0.00000 0.0 0.0 0.0 0.0

401964 rows × 34 columns

In [ ]:
merged.to_csv('../temp/merged.bed', sep='\t',index=None)
In [11]:
merged = pd.read_csv('../temp/merged.bed', sep='\t')
In [13]:
fig = sns.pairplot(merged[merged.columns[8:14]], corner=True, diag_kind="kde", kind="reg", plot_kws ={"scatter_kws":{"alpha":.05}})
def col_nan_scatter(x,y, **kwargs):
    df = pd.DataFrame({'x':x[:],'y':y[:]})
    df = df[df.sum(0)!=0]
    x = df['x']
    y = df['y']
    plt.gca()
    plt.scatter(x,y)
def col_nan_kde_histo(x, **kwargs):
    df = pd.DataFrame({'x':x[:]})
    df = df[df['x']!=0]
    x = df['x']
    plt.gca()
    sns.kdeplot(x)
fig = fig.map_upper(col_nan_scatter)
fig = fig.map_upper(col_nan_kde_histo)
fig.savefig('../results/cobinding/pairplot_experiments.pdf')
plt.show()
counts,val = np.unique(merged[merged.columns[8:]].astype(bool).sum(1).values, return_counts=True)
fig = sns.barplot(data=pd.DataFrame(val, index=counts,columns=['counts']).T).set_yscale("log")
fig.savefig('../results/cobinding/pairplot_experiments.pdf')
plt.show()
i = merged[merged.columns[8:]].astype(bool).sum(1)
print(i.max(),i.mean(),i.min())
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-13-100c054f8604> in <module>
     13     plt.gca()
     14     sns.kdeplot(x)
---> 15 fig = fig.map_upper(col_nan_scatter)
     16 fig = fig.map_upper(col_nan_kde_histo)
     17 fig.savefig('../results/cobinding/pairplot_experiments.pdf')

~/.local/lib/python3.8/site-packages/seaborn/axisgrid.py in map_upper(self, func, **kwargs)
   1385         """
   1386         indices = zip(*np.triu_indices_from(self.axes, 1))
-> 1387         self._map_bivariate(func, indices, **kwargs)
   1388         return self
   1389 

~/.local/lib/python3.8/site-packages/seaborn/axisgrid.py in _map_bivariate(self, func, indices, **kwargs)
   1496             y_var = self.y_vars[i]
   1497             ax = self.axes[i, j]
-> 1498             self._plot_bivariate(x_var, y_var, ax, func, kw_color, **kws)
   1499         self._add_axis_labels()
   1500 

~/.local/lib/python3.8/site-packages/seaborn/axisgrid.py in _plot_bivariate(self, x_var, y_var, ax, func, kw_color, **kwargs)
   1501     def _plot_bivariate(self, x_var, y_var, ax, func, kw_color, **kwargs):
   1502         """Draw a bivariate plot on the specified axes."""
-> 1503         plt.sca(ax)
   1504         if x_var == y_var:
   1505             axes_vars = [x_var]

~/.local/lib/python3.8/site-packages/matplotlib/pyplot.py in sca(ax)
    856             m.canvas.figure.sca(ax)
    857             return
--> 858     raise ValueError("Axes instance argument was not found in a figure")
    859 
    860 

ValueError: Axes instance argument was not found in a figure
In [14]:
counts,val = np.unique(merged[merged.columns[8:]].astype(bool).sum(1).values, return_counts=True)
fig = sns.barplot(data=pd.DataFrame(val, index=counts,columns=['counts']).T).set_yscale("log")

TODO: what would be the distribution in case they are completly random?

Correlation over consensus set

In [38]:
sns.clustermap(np.corrcoef(stats.zscore(merged[merged.columns[8:]].values.T, axis=1)), figsize=(20, 20), xticklabels=merged.columns[8:], yticklabels=merged.columns[8:]).ax_col_dendrogram.set_visible(False)
In [445]:
merged[merged.columns[:8]].to_csv('../temp/conscensus.bed',sep='\t',index=None, columns=None)
In [ ]:
cor = np.log2(1+cor)
a = cor.var(0).argsort()
subcor = cor[:,a[-TOTVAR:]]
In [ ]:
subcor.shape
In [ ]:
subcor = PCA(PCS).fit_transform(subcor)
In [ ]:
%matplotlib inline
cmaps = ['Greys', 'Purples', 'Blues', 'Greens', 'Oranges', 'Reds',
         'YlOrBr', 'YlOrRd', 'OrRd', 'PuRd', 'RdPu', 'BuPu',
         'GnBu', 'PuBu', 'YlGnBu', 'PuBuGn', 'BuGn', 'YlGn']
import seaborn as sns
from matplotlib import pyplot 
fig, ax = pyplot.subplots(figsize=(20,15))
sns.heatmap(ax=ax, data=subcor,xticklabels=False, cmap=cmaps[-4],
            yticklabels=[i.split('.')[-4].split('/')[-1] for i in bigwigs],
            cbar_kws={"orientation": "horizontal"},
            vmax=0.7).set_title('PCs of each binding profile over conscensus peakset')
#fig.savefig("temp/co_occupancy_matrix.png")
In [ ]:
fig.savefig('PCs_binding_peakset.png')

clustering

I have tried gaussian mixtures and Agglomerative clustering algorithm. Only the second can create a hierarchical clustering.

It seems that gaussian mixture makes more sense given the data we have, for now, is more "homogeneous".

I am still not so happy with the clustering. It can be because of the how much importance, outlier values and the high number of noisy values from locations with no peaks.

We can use similar methods to RNAseq to improve this (clamping values, log transform, first round of PCA..)

In [ ]:
labels = GaussianMixture(n_components=2, covariance_type='diag').fit_predict(subcor)
In [ ]:
names = np.array([i.split('.')[-4].split('/')[-1] for i in bigwigs])
sort = labels.argsort()
p = helper.plotCorrelationMatrix(data=cor[sort],
                            names=names[sort],
                            colors=labels[sort],
                            title="correlation between TF occupancy",
                            interactive=True)
In [ ]:
show(p)
In [ ]:
p = helper.scatter(TSNE(2,5).fit_transform(subcor),labels=names, colors=labels)
In [ ]:
show(p)
In [ ]:
sns.clustermap(subcor)

annotatePeaks

In [5]:
additional = {}
additional['activation'] = chiphelper.simpleMergedPeaks(mergedpeak[mergedpeak.tf.isin(["H3K27ac",'H3K79me2','H3K36me3','H3K4me3'])], window=10, mergedFold="max")
additional['repression'] = mergedpeak[mergedpeak.tf=='H3K27me3']
additional['IRF2BP2'] = mergedpeak[mergedpeak.tf=='IRF2BP2_FLAG']
additional['MED1'] = mergedpeak[mergedpeak.tf=='MED1']
additional['SMC1'] = mergedpeak[mergedpeak.tf=='SMC1']
additional['CTCF'] = mergedpeak[mergedpeak.tf=='CTCF']
additional['POLII'] = mergedpeak[mergedpeak.tf=='POLII']
0.9999921975578356
In [ ]:
#add enhancers at TSS in the matrix (promoters)
# add super enhancers
# add other enhancers
In [ ]:
# liftover enhancer assignements
# merge enhancer assigned to cobinding
In [ ]:
chiphelper.MakeSuperEnhancers()
In [6]:
for key, val in additional.items():
    merged[key] = chiphelper.putInConscensus(merged[merged.columns[:8]],val)
merged = merged.replace(np.nan,0)
0.00035326546655919434
/home/jeremie/.local/lib/python3.8/site-packages/numpy/core/fromnumeric.py:3334: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/jeremie/.local/lib/python3.8/site-packages/numpy/core/_methods.py:161: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
0.0054159078922490570.75537112776268525654
In [7]:
merged = merged.replace(np.nan,0)
In [8]:
merged.to_csv('../temp/merged.bed', sep='\t',index=None)
In [3]:
merged = pd.read_csv('../temp/merged.bed', sep='\t')

Co Binding Matrix

Look at AUC for all ChIPs over all peaks of all ChIPs

In [178]:
rand = np.random.choice(merged.index,5000)
viridis = cm.get_cmap('viridis', 256)
data = merged[merged.columns[-7:]]
for val in data.columns:
    data[val] =stats.zscore(np.log2(1+data[val]))
data = (((data -data.min(0))/ (data.max(0)))*256).astype(int)
data = data.loc[rand]
for val in data.columns:
    a = [viridis(v) for v in data[val]]
    data[val] = a
<ipython-input-178-7947baeca73b>:5: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  data[val] =stats.zscore(np.log2(1+data[val]))
In [185]:
fig = sns.clustermap(np.log2(1.01+merged[merged.columns[8:-7]].loc[rand].T),col_cluster=False, z_score=0, vmin=0,vmax=3, col_colors = data, figsize=(30,20),xticklabels=False)
fig.ax_col_dendrogram.set_visible(False)
fig.savefig('../temp/clustermap_cobinding_scaled_marks.pdf')
plt.show()
In [198]:
fig = sns.clustermap(np.log2(1.01+merged[merged.columns[8:-7]].loc[rand].T), vmin=0,vmax=3,figsize=(30,20),z_score=0,col_colors=data, xticklabels=False)
fig.ax_col_dendrogram.set_visible(False)
fig.savefig('../temp/clustermap_cobinding_scaled_marks.pdf')
plt.show()

Looking at peak overlap

How many of peak in A (column) overlaps with peak in B (rows)

in other words:

what is the percentage of B's peaks that are overlaped by A's peaks

In [90]:
correlation, overlap = chiphelper.computeOverlapCorrelation(merged, norm=True)
In [274]:
data = pd.DataFrame(data=overlap,index=merged.columns[8:], columns=merged.columns[8:])
link = linkage(data.iloc[:-7]) # D being the measurement
col = data[-7:]
col = col[[co for co in col.columns if co not in col.index.tolist()]]
for val in col.columns:
    a = [viridis(v) for v in col[val]]
    col[val] = a
fig = sns.clustermap(data.iloc[:-7][data.columns[np.concatenate((leaves_list(link),[26,27,28,29,30,31,32]))]], row_linkage=link, col_colors=col.T, col_cluster=False)
fig.ax_col_dendrogram.set_visible(False)
fig.savefig('../results/cobinding/overlap.pdf')
plt.show()

data on the experiments

In [193]:
pd.concat([merged[merged.columns[8:]].astype(bool).sum(0),
           merged[merged.columns[8:]].max(),
           merged[merged.columns[8:]].replace(0, np.NaN).mean(),
          merged[merged.columns[8:]].replace(0, np.NaN).var()],axis=1).rename(columns={0:'sum',1:'max',2:'mean',3:'std'})
Out[193]:
sum max mean std
ZEB2 14840 105.835243 6.598060 26.193602
PU1 132026 39.307407 6.302286 11.779725
MYBL2 352 35.384420 8.393023 18.988820
ZMYND8 22763 45.176055 7.383982 12.719919
IKZF1 35402 75.841090 6.598409 12.622733
SP1 25642 101.007463 12.236169 126.036058
RUNX1 42088 60.824327 8.767299 26.120400
PU1_FLAG 165175 64.507190 11.862644 110.706303
LDB1 569 30.134750 6.110416 5.419185
IRF8 64621 150.599790 16.782527 460.609597
CDK13 96 75.114920 6.871828 68.417806
GATA2 18069 191.226520 14.058436 268.840493
GFI1 7228 75.893420 8.781835 40.980955
CEBPA 192769 173.884930 7.735989 26.588283
MYC 46839 31.905473 4.947885 3.500217
LMO2 17979 49.521190 6.598640 12.381035
RUNX2 78155 75.824647 9.413992 60.833581
FOXP1 45717 82.752185 6.725721 17.156071
MYB 25772 125.953655 8.390215 49.654342
ETV6 6910 80.067493 9.697021 42.209236
MAX 94660 46.743730 5.207576 7.309308
MEIS1 63602 99.050070 9.602609 89.675024
FLI1 57711 132.340180 13.141390 168.313096
LYL1 12187 89.225010 8.921623 54.561422
MEF2D_FLAG 21507 72.611320 5.271724 10.712896
ELF2 41249 114.326750 11.153641 90.078763

Correlation only on overlaps

on the overlaps given above

In [273]:
data = pd.DataFrame(data=correlation,index=merged.columns[8:], columns=merged.columns[8:])
link = linkage(data.iloc[:-7]) # D being the measurement
col = data.iloc[-7:]
col = col[[co for co in col.columns if co not in col.index.tolist()]]
for val in col.columns:
    a = [viridis(v) for v in col[val]]
    col[val] = a
fig = sns.clustermap(data.iloc[:-7][data.columns[np.concatenate((leaves_list(link),[26,27,28,29,30,31,32]))]], row_linkage=link, col_colors=col.T, col_cluster=False)
fig.ax_col_dendrogram.set_visible(False)
fig.savefig("../results/cobinding/correlation_onoverlap.pdf")
plt.show()
In [ ]:
chiphelper.filterOnEnrichment(,toremove=[""])

TODO

plots during merge

  • plot a histogram (bincount) of the signal for each peaks:
    • peaks in A, B
    • unique peaks in A, B
    • peaks in merged version (avg)
    • scatter plot matrix of signal correlation between 1to1 for all
  • print average size in unique vs overlap peaks
  • ratios of unique peaks to total peaks for each TF

plots after merge

  • create a correlation plot on 1to1 overlap only.
  • histogram of number of peaks per amount of TF cobound
  • create a correlation matrix only on A's peakset
  • create a normalized overlap matrix (given the number of peaks and total peakset, what is the expected overlap and how far away are we from it) z-score norm
  • create a clustermap over peaks

improving merge

  • analyse only when peak is supported by two replicates
  • remove GSE1 and CDK13 from cobinding matrix
  • do quantile normalization over the signals (after setting up the zeroes)
  • removing bad chips:
    • look, when badchip only does not correlate with anything else..
  • do log2+1 transform

peak enrichment

  • use chromHMM
    • use the Chipseq data we have (RNApol2, CTCF...)
  • filter weird enrichments (not in H3k27ac? in H4me3..)
  • make super enhancer enrichment from superenhancer matrix
  • merge the clusters and display enrichment of each (in TF and other marks)

Next step:

  • do cobinding based on gene targeted from ABC-model and merge the same way as before (mean value); merge enrichments as well
  • look at data targetting CRC members
  • look for enrichment at the gene level (from gene annotations)
  • compute/refine binding patterns
    • use MEME
    • use DeepBind
  • look for difference in binding patterns to the litterature (db of binding patterns)
  • look for predictability of RNP expression change based on bound genes
    • do we see genes with opposite effects?
    • can it be explained by the set of co-bound TFs?
  • do dependency/expression prediction from all gene level features
    • look for most explanatory regressors
  • repeat the process across all samples with H3K27ac+RNAseq data we have.
    • call mutations from H3k27ac data
    • MEME analysis of likely bound TFs, except if TF is not expressed
    • compute enrichments
    • from same gene assignements, as found on the general consensus peak set, can we find good dependency/expression prediction
  • does location

questions:

  • how did andrew defined significant peaks?

based on closest expressed gene

In [ ]:
chiphelper.AssignToClosestExpressed()

based on the ABC model

They tested a new model based on and validated by CRISPRi-FlowFISH which is basically able to find enhancer mapping to genes. They used it to compute their model's Accuracy and found a 70% accuracy compared to less than 50% for closest expressed gene.

Way to integrate our HiC data (need ATAC-seq like data as well, but openly available)

In [ ]:
Helper.scatter(TSNE(2,5).fit_transform(data.T), labels=zones.columns[11:],colors=labels)

Compute the CRC and merge with gene assignement

~1500 lines of code. Seems to be a slightly updated version from the code I used the first time which was from their originial CRC paper.

There is not a lot of documentation but it was just updated last week and might continue like that

find set of TFs that explain the most cobindings

relate to gene depencies

what else?

Compare data with other labs (H3K27, HiC..)

we need to redo everything for similar normal cell type, getting TFs based on the CRC (find it with CRCmapper or on litterature)